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Program Description 1 


Project CRISS ® (CReating Independence through Student- 
owned Strategies) is a professional development program for 
teachers 2 that aims to improve reading, writing, and learning 
for 3rd- through 12th-grade students. The implementation of 
Project CRISS ® does not require a change in the curriculum or 
materials being used in the classroom, but instead calls for a 
change in teaching style to focus on three primary concepts 
derived from cognitive psychology and brain research. These 
three concepts include students (1) monitoring their learning 
to assess when they have understood content, (2) integrating 
new information with prior knowledge, and (3) being actively 


involved in the learning process through discussing, writing, 
organizing information, and analyzing the structure of text to 
help improve comprehension. 

In Project CRISS®, teachers incorporate these concepts into 
their regular classroom instruction through the use of com- 
prehension strategies (such as using background knowledge, 
questioning, organizing graphically, and summarizing). Project 
CRISS® calls for students to apply these comprehension strate- 
gies to content they encounter, to gain an understanding of when 
and how it is most appropriate to use these strategies, and to 
learn to use the strategies that work best for them. 


Research 3 


Two studies of Project CRISS ® that fall within the scope of 
the Adolescent Literacy review protocol meet What Works 


Clearinghouse (WWC) evidence standards. The two studies 
included 2,569 students, ranging from grade 4 through grade 



1. The descriptive information for this program was obtained from a publicly available source: the developer’s website (http://www.projectcriss.com, 
downloaded October 2009). The WWC requests developers to review the program description sections for accuracy from their perspective. Further 
verification of the accuracy of the descriptive information for this program is beyond the scope of this review. The literature search reflects documents 
publicly available by August 2009. 

2. Project CRISS ® also has several other training programs and support materials available, including: (1) CRISS for Administrators, which is designed 
to provide guidance to administrators on how to plan, implement, and maintain Project CRISS ® in a school or district; (2) CRISS for Students, which is 
designed to teach CRISS principles and strategies directly to 6th- through 9th-grade students; (3) CRISS for Parents, which is designed to acquaint 
parents with CRISS principles and strategies; and (4) CRISS for Homeschool Parents, which is designed to help parents that are home schooling their 
children incorporate CRISS strategies and principles into their instruction. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Handbook, Chapter III), 
as described in protocol version 2.0. 
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Research (continued) 6, who attended public schools in Arizona, California, Florida, 
Georgia, Louisiana, Montana, Oregon, Texas, Virginia, and 
Wisconsin. 4 

Based on these two studies, the WWC considers the extent 
of evidence for Project CRISS® on adolescent learners to be 



medium to large for the comprehension domain. No studies that 
meet WWC evidence standards examined the effectiveness of 
Project CRISS® on adolescent learners in the alphabetics, read- 
ing fluency, or general literacy achievement domains. 



Effectiveness 



Project CRISS® was found to have potentially positive effects on comprehension for adolescent learners. 





Reading 




General reading 


Alphabetics 


fluency 


Comprehension 


achievement 



Rating of effectiveness 


na 


na 


Potentially positive effects 


na 


Improvement index 5 


na 


na 


Average: +20 percentile points 


na 




na 


na 


Range: -2 to +38 percentile points 


na 



na = not applicable 



Absence of conflict of 
interest 



One of the studies in this intervention report, James-Burdumy et 
al. (2009), was prepared, in-part, by staff of Mathematica Policy 
Research. For this reason, and because the principal investigator 
for the WWC review of adolescent literacy was also a lead author 



of this study, the study was rated by researchers unaffiliated with 
Mathematica, who also prepared the intervention report. The 
report was then reviewed by the principal investigator, a WWC 
Quality Assurance reviewer, and an external peer reviewer. 



Additional program 
information 



Developer and contact 

Project CRISS® was originally developed in the late 1970s by Dr. 
Carol Santa and a team of school teachers from Kalispell School 
District #5 in Montana. More recently, the development of Project 
CRISS® has been led by Lynn Havens, former Kalispell second- 
ary math and science teacher. Address: Project CRISS , 40 
Second Street East, Suite 249, Kalispell, MT 59901. Email: info@ 
projectcriss.com. Web: http://www.projectcriss.com/. Telephone: 
(877) 502-7477 (toll free), (406) 758-6440 (direct). 

Scope of use 

According to the developers, Project CRISS® has been used 
across the curriculum in elementary classrooms and in middle 
and high school math, science, social studies, language arts, 



fine arts, technology, and physical education classes. It has been 
used with students of all abilities in both urban and rural settings. 

Teaching 

Project CRISS® employs a teaching and learning process in 
which teachers model strategies for students and provide 
time for guided practice, with the goals of helping students (1) 
understand their learning processes and content, and (2) transfer 
strategies to independent learning situations. 

The CRISS training introduces teachers to the CRISS 
Strategic Learning Plan, which is intended to guide selection of 
content, setting of learning goals and objectives, assessment 
of student learning, and planning of instruction. The training is 
designed to instruct participants in ways to help their students 
interact with content, understand patterns and structures of 



4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

5. These numbers show the average and range of student-level improvement indices for all findings across the studies. 
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Additional program text, engage actively in the learning process, write reports and 
information (continued) essays, and learn new vocabulary. The training also addresses 

ways teachers can help students become more reflective 
(metacognitive) about their learning processes. Participants in 
Project CRISS® workshops receive a teacher resource guide that 
is designed to assist them in incorporating CRISS principles into 
their classroom instruction. 

Cost 

Project CRISS® offers two levels of workshops for teachers. 

Level I training, which is 12 to 24 hours, is designed to prepare 
teachers to incorporate CRISS principles and strategies into 
the classroom. Costs for Level I training— which typically range 
from $50 to $200 per participant— depend on whether the 



training takes place in the teachers’ own district or another 
district, and whether it is conducted by a national trainer, who 
receives a $1,000 per day honorarium. 6 Level II training, which 
is a minimum of 28 hours over a 4-day period, is designed to 
prepare experienced CRISS teachers to become CR/SS-certified 
trainers who can take on more CR/SS-related support and train- 
ing responsibilities. 7 The cost for materials— which ranges from 
$250 to $700 per participant— depends on whether teachers are 
trained in their own district or another district. 6 The training is 
facilitated by a CRISS Master Trainer (additional costs include a 
$1,000 per day honorarium and travel expenses). Implementation 
support options are available, including administrator training, 
collaborative learning team tools, parent workshops, and tailor- 
made workshops. 



Research 



Thirty-one studies reviewed by the WWC investigated the effects 
of Project CRISS ® on adolescent learners. Two studies (Horsfall 
& Santa, 1994, and James-Burdumy et al., 2009) are random- 
ized controlled trials that meet WWC evidence standards. 

The remaining 29 studies do not meet either WWC evidence 
standards or eligibility screens. 

Meets evidence standards 

Horsfall and Santa (1994) conducted a random assignment study 
of Project CRISS ® in 4th-, 6th-, 8th-, and llth-grade classrooms 
across three school settings: (1) rural Montana, (2) a working 
class community in central Florida, and (3) suburban Virginia. 
Teachers within each school were randomly assigned either to 
Project CRISS® or to a regular instruction control condition. The 
WWC based its effectiveness ratings on comparisons of 120 stu- 
dents attending six Project CRISS ® classrooms and 111 students 



attending six control classrooms. 8 The study measured changes 
in comprehension outcomes after one semester (approximately 
18 weeks) of program participation. 

James-Burdumy et al. (2009) conducted a randomized 
controlled trial that examined the effects of Project CRISS® (as 
well as three other reading comprehension curricula) on compre- 
hension of 5th-grade students across the United States. Authors 
randomly assigned 89 schools in ten geographically diverse low 
income school districts either to one of four curricula: (1) Project 
CRISS®, (2) ReadAbout, (3) Read for Real, and (4) Reading for 
Knowledge, or to a control condition that had no access to 
these curricula. The WWC based its effectiveness ratings for the 
Project CRISS® portion of this analysis on 1,155 students attend- 
ing 17 Project CRISS® schools and 1,183 students attending 21 
comparison schools. The study measured reading comprehen- 
sion outcomes after nine months of program implementation. 



6. For more detailed information on the costs of CRISS trainings and support materials, consult: http://www.projectcriss.eom/costs.php#levell. 

7. The Level II trainings are designed to provide four levels of CRISS certification, ranging from Facilitator certification, which allows a teacher to provide 
follow-up support to CRISS teachers in their district, up to a Master Trainer certification, which allows one to conduct Level I and II trainings anywhere in 
the United States. 

8. Only the 4th and 6th grade samples are included in the calculations the WWC used to rate the effectiveness of Project CRISS®. The 8th and 11th grade 
samples are excluded from the WWC ratings of effectiveness because the measures of effectiveness cannot be attributed solely to the intervention— 
there was only one teacher assigned to Project CRISS ® and one teacher assigned to the control condition in the 8th and 11th grade samples. 
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Research (continued) 



Effectiveness 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or medium to large (see the WWC Procedures and Stan- 
dards Handbook, Appendix G). The extent of evidence takes into 
account the number of studies and the total sample size across 
the studies that meet WWC evidence standards with or without 
reservations. 9 



The WWC considers the extent of evidence for Project 
CRISS® to be medium to large for the comprehension domain 
for adolescent learners. No studies that meet WWC evidence 
standards with or without reservations examined the effective- 
ness of Project CRISS® on adolescent learners in the alphabet- 
ics, reading fluency, or general literacy achievement domains. 



Findings 

The WWC review of interventions for Adolescent Literacy 
addresses student outcomes in four domains: alphabetics, read- 
ing fluency, comprehension, and general literacy achievement. 
The studies included in this report cover one domain: compre- 
hension (reading comprehension construct). The findings below 
present the authors’ estimates and WWC-calculated estimates 
of the size and the statistical significance of the effects of Project 
CRISS ® on adolescent learners. 10 

Comprehension. Two studies reviewed findings in the 
comprehension domain. Horsfall and Santa (1994) reported, and 
WWC calculations confirmed, that students in the Project CRISS® 
condition demonstrated significantly greater gains in a staff-devel- 
oped CRISS free-recall assessment than comparison students in 
the 4th and 6th grades. 11 James-Burdumy et al. (2009) did not find 
any statistically significant effects of Project CRISS® on the pas- 
sage comprehension subtest of the Group Reading Assessment 
and Diagnostic Evaluation or either the science or social studies 



reading comprehension assessments. The WWC-calculated 
average effect across measures was not statistically significant or 
large enough to be considered substantively important according 
to WWC criteria (that is, an effect size of at least 0.25). 

In summary, for the comprehension domain, one study 
showed a statistically significant positive effect, and one study 
showed indeterminate effects. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research design, 
the statistical significance of the findings, the size of the difference 
between participants in the intervention and the comparison condi- 
tions, and the consistency in findings across studies (see the WWC 
Procedures and Standards Handbook, Appendix E). 



9. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for Project CRISS ® is in Appendix A5. 

10. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of 
Horsfall and Santa (1994), corrections for clustering and multiple comparisons were needed, so the significance levels may differ from those reported in 
the original study. In the case of James-Burdumy et al. (2009), the authors adjusted for clustering and applied a correction for multiple comparisons. 

11. The 8th- and llth-grade samples are excluded from the WWC ratings of effectiveness because the measures of effectiveness cannot be attributed 
solely to the intervention— there was only one teacher assigned to Project CRISS ® and one teacher assigned to the control condition at each grade level. 
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The WWC found Project 
CRISSP to have potentially 
positive effects on 
comprehension for 
adolescent learners 
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Appendix A1.1 Study characteristics: Horsfall & Santa, 1994 (random assignment study) 



Characteristic 


Description 


Study citation 


Horsfall, S., &Santa, C. (1994). Project CRISS: Validation report for the Program Effectiveness Panel. Unpublished manuscript. 


Participants 


Sixteen intact classrooms of students in grades 4, 6, 8, and 11 participated in the study during the 1991-92 school year; however, only analysis of grades 4 and 6 were 
included in this review. 1 Teachers within each of three schools were randomly assigned either to Project CRISS ® or to the control condition. Teachers assigned to the interven- 
tion received CRISS training; control group teachers did not. Within each grade level, 4 and 6, there were three classrooms assigned to Project CRISS ® and three classrooms 
assigned to the control group. Four or five students in each class were excluded from analyses due to attrition; there was no attrition of teachers. In all, the analysis sample 
consisted of 120 students attending six Project CfflSS® classrooms and 111 students attending six control group classrooms. 


Setting 


The study took place across three different settings: (1) Kalispell School District, MT, a rural district in northwestern Montana that serves primarily white students; (2) Putnam 
County School District, FL, a district in central Florida that serves a population composed of white (77%), black (20%), and Hispanic students; and (3) Stafford School District, 
VA, a district in suburban Washington, DC that serves primarily white students. 


Intervention 


Intervention group students received Project OTSS® strategies as part of their regular instruction for approximately 1 8 weeks during one semester. 


Comparison 


Control group students received regular instruction and were not given Project CRISS ® strategies. 


Primary outcomes 
and measurement 


For both the pretest and posttest, students took the staff-developed “free recall” tests that require students to remember details from a passage read the day before. For a 
more detailed description of this outcome measure, see Appendix A2. 


Staff/teacher training 


Teachers assigned to the intervention received Project CRISES’ training. Districts selected a local facilitator to coordinate the program. The facilitator organized a 12-hour 
training conducted over two consecutive days. During this training, trainers modeled Project CRISS ® strategies, and teachers were given the opportunity to apply each of the 
Project CRISES strategies to their own curriculum materials. After teachers completed the training, the facilitator worked with project staff to set up a follow-up session three 
to six months after the completion of the final training session. Teachers frequently met to share Project CRISSS ideas before or after school or during duty-free periods. In 
addition, Project CRISS ® trainers provided follow-up assistance for teachers through on-site visits, demonstration lessons, newsletters, and a computer network. 



1. For the 8th and 11th grade samples, there was only one teacher assigned to the treatment group and one teacher assigned to the control group for each grade level. For this reason, the results 
from the 8th and 11th grade analyses could be confounded with factors unrelated to the CRISS treatment and thus are not included in the WWC’s rating of effectiveness. 
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Appendix A1.2 Study characteristics: James-Burdumy et al., 2009 (randomized controlled trial) 



Characteristic 


Description 


Study citation 


James-Burdumy, S., Mansfield, W., Deke, J., Carey, N., Lugo-Gil, J., Hershey, A., et al. (2009). Effectiveness of selected supplemental reading comprehension interventions: 
Impacts on a first cohort of fifth-grade students (NCEE 2009-4032). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. 


Participants 


The study, which explored the impact of Project CRISES as well as three other reading comprehension curricula ( ReadAbout , Read for Real, and Reading for Knowledge), 
included 6,350 5th-grade students from 89 schools in ten school districts. Districts that had at least 12 Title 1 schools and who were not implementing any of the four selected 
curricula were recruited into the study. Within each school district, schools were randomly assigned either to one of the four intervention conditions or to the control group. 
Eligible students attended study schools and were enrolled in grade 5 when baseline tests were administered or transferred in after baseline and before January 1 , 2007. 
Multiage grade levels and non-mainstreamed special education students were excluded from the sample. The analysis that is included in this review focused on the effect of 
Project CRISS® and examined a sample of 1,155 students attending 17 Project CRISS® schools and 1,183 students attending 21 control schools. 


Setting 


The study took place in ten geographically diverse school districts in eight states (Arizona, California, Florida, Georgia, Louisiana, Oregon, Texas, and Wisconsin). To be eligible 
for the study, school districts needed to have (1) at least 12 schools that received Title 1 funds, (2) at least 40% of students eligible for the federal free or reduced-price lunch 
program, and (3) at least 60 5th-grade students per school. The school districts in the study were significantly larger, more disadvantaged, and more urban than the average 
U.S. school district. 


Intervention 


Intervention group students received Project CRISS ® strategies as part of their regular instruction. Instructional components included: (1) use of student and teacher editions 
of Learning How to Learn, which provided detailed lesson plans, learning, and practice through use of Tough Terminators, a science trade book; (2) use of a variety of graphic 
organizers and note-taking, discussion, vocabulary, and writing strategies; and (3) application of strategies to regular science and social studies texts. Project CRISS® 
teachers, on average, were observed engaging in 78% of teaching practices important to intervention implementation. The study reported students’ reading comprehension 
outcomes after nine months of program implementation. 


Comparison 


Control group schools did not have access to any of the four curricula being tested. Control group teachers could, however, use other supplemental reading programs. 


Primary outcomes 
and measurement 


For the pretest, students took the passage comprehension subtest of the Group Reading Assessment and Diagnostic Evaluation (GRADE) and the Test of Silent Contextual 
Reading Fluency (TOSCRF). For the posttest, all students took the passage comprehension subtest of the GRADE. Students were also randomly assigned to take one of two 
reading comprehension assessments developed by the Educational Testing Service (ETS) for this study; these tests focused on either science or social studies. For a more 
detailed description of these outcome measures, see Appendix A2. 


Staff/teacher training 


Project OTSS® teachers received 18 hours of initial training, including 12 hours on using the strategies in the teacher’s guide and six hours on using the student text and 
workbook. Teachers received a training manual, a teacher's guide, a student text, and a wrap-around edition of the student workbook. In addition, teachers received six hours 
of follow-up training. Trainers also visited schools monthly to observe teachers and provide feedback. The developer also encouraged teachers to use bi-weekly study teams in 
which teachers review and discuss their use of CRISS strategies. 
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Appendix A2 Outcome measures for the comprehension domain 



Outcome measure 


Description 


Reading comprehension construct 


Staff-developed (CRISS) 
free recall assessment 


The outcome is a staff-developed “free recall” measure that requires students to read a passage (2-4 pages, depending on grade level) over a 40-minute period; then, 24 
hours later, students write down from memory as much as they can remember from the passage. Students are scored based on the number of idea units (one point per idea) 
they remember. The topics varied across grade levels, and care was taken to choose readings that had a content base similar to what students would experience in their 
regular coursework but with actual topics that would not normally have been covered in those courses. The Grade 4 assessment covered The Western Movement (770 words, 
2 single-spaced pages), and the Grade 6 assessment covered The Mystery of Thirst (920 words, 3 single-spaced pages). Inter-rater reliability ranged from 0.93 to 0.95 (as 
cited in Horsfall & Santa, 1994). 


Group Reading Assessment 
and Diagnostic Evaluation 
(GRADE)-Passage 
Comprehension subtest 


This standardized measure is a norm-referenced diagnostic test for all reading abilities. The Passage Comprehension subtest measures a student's understanding of an 
extended text through explicit and implicit multiple choice questions requiring questioning, predicting, summarizing, and clarifying information from several paragraphs (as 
cited in James-Burdumy et at, 2009). 


Educational Testing Service 
(ETS) science reading 
comprehension assessment 


This assessment, designed by the ETS, focuses on students' reading comprehension of science text. The test measures the ability to comprehend five science-related exposi- 
tory text passages based on responses to six multiple choice questions per passage. Internal consistency was reported as 0.85 (as cited in James-Burdumy et al., 2009). 


ETS social studies reading 
comprehension assessment 


This assessment, designed by the ETS, focuses on students' reading comprehension of social studies text. The test measures the ability to comprehend five social studies- 
related expository text passages based on responses to six multiple choice questions per passage. Internal consistency was reported as 0.84 (as cited in James-Burdumy et 
at, 2009). 
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Appendix A3 Summary of study findings included in the rating for the comprehension domain 1 









Authors’ findings from the study 














Mean outcome 
(standard deviation) 2 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms or 
schools/students) 


Project 

CRISS® Comparison 

group group 


Mean 

difference 3 
(Project CRISS® - 
comparison) 


Statistical 

Effect significance 5 

size 4 (at a = 0.05) 


Improvement 

index 6 



Horsfall & Santa, 1994 7 ' 8 

Free recall assessment Grade 4 6 classrooms/ 8.97 5.32 3.65 1.17 

118 students (3.38) (2.73) 

Free recall assessment Grade 6 6 classrooms/ 12.54 7.93 4.61 0.96 

113 students (5.35) (4.07) 

Average for comprehension (Horsfall & Santa, 1994) 9 1.07 

James-Burdumy et al., 2009 10 



GRADE-Passage 

comprehension 


Grade 5 


38 schools/ 
2,332 students 


100.48 

(14.20) 


101.06 

(13.69) 


-0.57 


-0.04 


ns 


-2 


ETS science comprehension 


Grade 5 


38 schools/ 


501.44 


500.76 


0.69 


0.02 


ns 


+1 






1,153 students 


(29.51) 


(27.59) 










ETS social studies 


Grade 5 


38 schools/ 


499.64 


500.61 


-0.96 


-0.03 


ns 


-1 


comprehension 




1,140 students 


(30.57) 


(29.68) 










Average for comprehension (James-Burdumy et al., 2009) 9 








-0.02 


na 


-1 


Domain average for comprehension across all studies 9 








0.53 


na 


+20 



ns = not statistically significant 
na = not applicable 

GRADE = Group Reading Assessment and Diagnostic Evaluation 
ETS = Educational Testing Service 

1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the comprehension domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 



Statistically 

significant 


+38 


Statistically 

significant 


+33 


na 


+36 



(continued) 
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Appendix A3 



Summary of study findings included in the rating for the comprehension domain 1 (continued) 



6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Horsfall and Santa (1994), corrections for clustering and multiple comparisons were needed, so the significance levels may differ 
from those reported in the original study. In the case of James-Burdumy et al. (2009), the authors adjusted for clustering and applied a correction for multiple comparisons. 

8. The Project CRISS® group mean outcome values for Horsfall and Santa (1994) are the unadjusted control group posttest means plus the difference in mean gains between the intervention and 
control groups. Control group means are unadjusted. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 

10. Sample sizes, regression-adjusted means, and standard deviations were provided to the WWC by the study authors and thus differ slightly from the information presented in the original study. 



WWC Intervention Report Project CRISS® 



June 2010 



12 




Appendix A4 Project CRISSP rating for the comprehension domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the comprehension outcome domain, the WWC rated Project CRISS® as having potentially positive effects for adolescent learners. The remaining ratings (mixed 
effects, no discernible effects, potentially negative effects, or negative effects) were not considered, as Project CRISS ® was assigned the highest applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study showed statistically significant positive effects. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No studies showed statistically significant or substantively important negative effects, and one study showed indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Only one study showed statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence 1 


Alphabetics 


na 


na 




na 


na 


Reading fluency 


na 


na 




na 


na 


Comprehension 


2 


41 




2,569 2 


Medium to large 


General literacy achievement 


na 


na 




na 


na 



na = not applicable/not studied 



1. A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Other- 
wise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 

2. This sample size varies slightly from the sample sizes presented in Appendix A3 because in James-Burdumy et al. (2009), the total sample size includes students who had outcomes for at least 
one of the three tests. 
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